Augmented Reality (AR) may be realized by overlaying images of virtual objects onto an image of the real world environment including one or more real world objects. The images of virtual objects are positioned in a user's Field of View (FOV) to give the virtual objects an apparent location in the environment. To ensure that the images of the virtual objects are displayed in appropriate positions in the user's FOV relative to the viewed locations of the real world objects in the environment, various AR applications may anchor the virtual objects in the environment based on the location of one or more real world objects. When the user is presented with a distorted view of the environment, however, the AR application may have difficulty arranging virtual objects in accordance with the distorted view. As a result, the AR application may anchor and present virtual objects at positions inconsistent with the AR experience otherwise provided to the user, resulting in a degraded AR experience.